System to detect objects ejected from a vehicle

ABSTRACT

A method for detecting objects ejected from a cabin of a vehicle includes generating window image data of a vehicle window opening of the vehicle with a camera mounted on the vehicle; and processing the window image data with a controller operably connected to the camera to generate ejected object data corresponding to at least one object ejected from the cabin through the vehicle window opening. The method further includes associating the ejected object data with occupant data of an occupant of the vehicle with the controller.

FIELD

This disclosure relates to the field of vehicles and computer visionand, in particular, to identifying objects, such as waste and litter,ejected from a vehicle using computer vision.

BACKGROUND

Ride hailing services and ride sharing services are increasing inpopularity as a means of transportation for people in urbanenvironments. Moreover, in connection with autonomous vehicletechnology, self-driving taxis and shuttles are envisioned in the nearfuture. Ride sharing services and the prospect of shared autonomousvehicles have resulted in new business models and correspondingchallenges.

One of the challenges in maintaining a fleet of shared vehicles, bothhuman driven and autonomously controlled, is keeping the vehicles cleanand free from the waste of previous occupants. For example, some sharedvehicles use cameras or other systems to identify waste left behindinside the vehicle cabin by an occupant. The occupant may be assessed aservice charge for cleaning the vehicle if the occupant leaves thevehicle in a sufficiently dirty condition.

In order to avoid the service charge when utilizing a shared vehicle,most occupants clean up after themselves by disposing of their waste ina proper receptacle, such as a waste bin. Other occupants, however, takea different approach to avoid assessment of the service charge. Inparticular, some occupants eject waste and debris from the vehicle whilethe vehicle is in motion. Such an approach leaves the vehicle cabinclean and free of debris, but results in an untended consequence ofincreased litter along roadways and in neighborhoods.

The current business model for keeping shared vehicles cleanincentivizes some occupants to dispose of waste in an unintended way. Asa result, further developments in the area of shared vehicles andautonomous vehicles are desired as an effort to protect the environment,to save costs on communities for cleanup, and/or to provide overallcleaner neighborhoods and surroundings.

SUMMARY

According to an exemplary embodiment of the disclosure, a method fordetecting objects ejected from a cabin of a vehicle includes generatingwindow image data of a vehicle window opening of the vehicle with acamera mounted on the vehicle; and processing the window image data witha controller operably connected to the camera to generate ejected objectdata corresponding to at least one object ejected from the cabin throughthe vehicle window opening. The method further includes associating theejected object data with occupant data of an occupant of the vehiclewith the controller.

According to another exemplary embodiment of the disclosure, a methodfor detecting objects ejected from a cabin of a vehicle includesgenerating window image data of a vehicle window opening when acorresponding movable window panel is in an open position, andprocessing the window image data to generate ejected object datacorresponding to objects ejected from the cabin through the vehiclewindow opening. The method further includes generating littering eventdata based on the generated ejected object data, and associating thelittering event data with an operator of the vehicle.

According to a further exemplary embodiment of the disclosure, a systemfor detecting objects ejected from a cabin of a vehicle includes acamera and a controller. The camera is mounted on the vehicle and isconfigured to generate window image data of a vehicle window opening.The controller is operably connected to the camera to receive the windowimage data. The controller is configured (i) to process the window imagedata to generate ejected object data corresponding to at least oneobject ejected from the cabin through the vehicle window opening, and(ii) to associate the ejected object data with occupant data of anoccupant of the vehicle.

BRIEF DESCRIPTION OF THE FIGURES

The above-described features and advantages, as well as others, shouldbecome more readily apparent to those of ordinary skill in the art byreference to the following detailed description and the accompanyingfigures in which:

FIG. 1 is a top view block diagram illustrating an exemplary embodimentof a vehicle including a system for detecting objects ejected from acabin of a vehicle, as disclosed herein;

FIG. 2 is a side view block diagram illustrating the vehicle and systemof FIG. 1, and illustrating an occupant ejecting an object from thevehicle while the vehicle is in motion through a front window panel inan open window state;

FIG. 3 is a block diagram illustrating the vehicle and the system ofFIG. 1; and

FIG. 4 is a flowchart illustrating an exemplary method of operating thesystem of FIG. 1 in order to detect objects ejected from the cabin ofthe vehicle.

DETAILED DESCRIPTION

For the purpose of promoting an understanding of the principles of thedisclosure, reference will now be made to the embodiments illustrated inthe drawings and described in the following written specification. It isunderstood that no limitation to the scope of the disclosure is therebyintended. It is further understood that this disclosure includes anyalterations and modifications to the illustrated embodiments andincludes further applications of the principles of the disclosure aswould normally occur to one skilled in the art to which this disclosurepertains.

Aspects of the disclosure are described in the accompanying description.Alternate embodiments of the disclosure and their equivalents may bedevised without parting from the spirit or scope of the disclosure. Itshould be noted that any discussion herein regarding “one embodiment”,“an embodiment”, “an exemplary embodiment”, and the like indicate thatthe embodiment described may include a particular feature, structure, orcharacteristic, and that such particular feature, structure, orcharacteristic may not necessarily be included in every embodiment. Inaddition, references to the foregoing do not necessarily comprise areference to the same embodiment. Finally, irrespective of whether it isexplicitly described, one of ordinary skill in the art would readilyappreciate that each of the particular features, structures, orcharacteristics of the given embodiments may be utilized in connectionor combination with those of any other embodiment discussed herein.

For the purposes of the disclosure, the phrase “A and/or B” means (A),(B), or (A and B). For the purposes of the disclosure, the phrase “A, B,and/or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, Band C).

The terms “comprising,” “including,” “having,” and the like, as usedwith respect to embodiments of the disclosure, are synonymous.

As shown in FIG. 1, a vehicle 100 includes a system 104 configured todetect when an occupant 108 of the vehicle 100 ejects an object 112 fromthe vehicle 100 while the vehicle 100 is in motion. The system 104determines when occupant 108 litters by detecting, using computervision, when the occupant 108 moves items from within the vehicle 100 tooutside of the vehicle 100 through window openings 116 (FIG. 2) whilethe vehicle 100 is in motion. In one embodiment, the system 104 displaysa message to the occupant 108 of the vehicle 100 to instruct theoccupant 108 to stop the littering activity. The system 104 alsoassociates the object 112 with the identity of the occupant 108, therebyenabling the business operating the vehicle 100 to follow up with theoccupant 108 in an appropriate manner, such as by assessing a servicecharge against the occupant or by preventing the occupant 108 from againutilizing the vehicle 100. The litter sensing system 104 provides anincentive for occupants 108 of the vehicle 100 to dispose of waste (i.e.the object 112) in a proper receptacle instead of tossing the object 112on the roadway and in neighborhoods, so as to avoid the service charge.Each aspect of the system 104 and a method 400 (FIG. 4) of operating thesystem 104 are described herein.

With reference to FIGS. 1 and 2, an exemplary vehicle 100 including thesystem 104 is illustrated. The vehicle 100, in one embodiment, is ashared vehicle 100 that may be occupant 108 controlled (i.e. levels zerothrough four autonomous control) or fully autonomously controlled (i.e.level five autonomous control). In other embodiments, the vehicle 100 isa rental car, a shuttle, a limousine, a corporate vehicle, a liveryvehicle, a taxi, or a personal vehicle. The vehicle 100 is any vehiclethat carries human occupants 108.

The vehicle 100 includes a chassis 120 having doors 124 and windowpanels 128, and defining an interior cabin 132. As shown in FIG. 2, theleft front door 124 defines the window opening 116 in which the movablewindow panel 128 is positionable in an open window state and a closedwindow state. In FIG. 2, the left front window panel 128 is in the openwindow state and the left rear window panel 128 of the left rear door124 is in the closed window state. In some embodiments, the vehicle 100includes a further window opening 116 and a corresponding further windowpanel 128 located in a ceiling of the vehicle 100 and referred to as asunroof or a moonroof. The sunroof and/or moonroof is included in thewindow opening 116 and the window panel 128.

As shown in FIG. 1, the vehicle 100 further includes occupant seats 140located within the cabin 132, a dashboard 144 including a display screen148, at least one speaker 152, a center console 156, side mirrorhousings 160, and an electronic control unit (“ECU”) 164. In theexemplary vehicle 100, four occupant seats 140 are included. In otherembodiments, the vehicle 100 may include any desired number of occupantseats 140 in any configuration.

The dashboard 144 is located at the front of the vehicle 100 andincludes instruments and controls for operating and/or interacting withthe vehicle 100.

With continued reference to FIG. 1, in one embodiment, the displayscreen 148 is integrated into the dashboard 144. The display screen 148displays vehicle data to the occupants 108 of the vehicle 100. In anoccupant-controlled vehicle 100, the display screen 148 displays vehiclespeed, remaining charge, navigation information (from a globalpositioning system “GPS”), and other messages to the driver 108 and theother occupants 108 of the vehicle 100. In an autonomous vehicle 100,the display screen 148 displays information to the occupants 108 of thevehicle 100, such as the time remaining to destination and instructionsfor interacting with the vehicle 100. The display screen 148 isconfigurable to display any desired information to the occupants 108 ofthe vehicle 100. In one embodiment, the display screen 148 is providedas a liquid crystal display or any other information display technology.

The at least one speaker 152 is at least partially located in the cabin132. The speaker 152 generates sound to provide auditory information andentertainment to the occupant 108.

The center console 156 is located between the front two occupants seats140. The center console 156 may include a space for holding occupantpossessions, such as smartphones, wallets, and/or beverage containers(i.e. object 112).

The side mirror housings 160 are mounted on the vehicle 100 and includea mirror (not shown) positioned to provide a view along the side and tothe rear of the vehicle 100. In some embodiments, such as a fullyautonomous vehicle 100, the side mirror housings 160 include cameras,such as the mirror cameras 208 described below, instead of mirrors.

The ECU 164 is configured to control electronic components of thevehicle 100. For example, the ECU 164 is configured to control aninternal combustion engine (not shown) and/or a traction motor (notshown) to result in desired movement of vehicle 100. In one embodiment,the ECU 164 is connected to at least one window state sensor 170 thatmonitors a status of the movable window panels 128, and to at least onedoor state sensor 174 that monitors a status of the doors 124. The ECU164 is also connected to a speed sensor 178 that monitors a speed ofmovement of the vehicle 100. The ECU 164 generates data corresponding tothe position of each movable window panel 128 (i.e. open or closed), thestatus of each door 124 (i.e. open or closed), and the speed of thevehicle 100. The ECU 164, in one embodiment, is electrically connectedto the window state sensor 170, the door state sensor 174, and the speedsensor 178 by a CAN bus (controller area network bus).

As shown in FIGS. 1 and 3, the litter sensing system 104 is mounted onthe vehicle 100 and includes cabin cameras 204, mirror cameras 208, atransceiver 212, and a memory 216 each operatively connected to acontroller 220. The cabin cameras 204, which are mounted on the vehicle100 within the cabin 132, are digital imaging devices configured togenerate cabin image data 228. The cabin cameras 204 are positioned toobserve the behavior and movements of the occupant 108 and to generatethe cabin image data 228 based on the behavior and movements of theoccupant 108. The cabin cameras 204 may be configured as visible lightcameras, thermal cameras, and/or infrared cameras.

The front cabin camera 204 (i.e. the cabin camera 204 closest to thedashboard 144 in FIG. 1) defines a field of view 232 (FIG. 1) thatincludes at least the entire front occupant seats 140, the dashboard144, the center console 156, and at least a portion of an interior floor236 of the vehicle 100. The front cabin camera 204, in one embodiment,is mounted on the dashboard 144 and in another embodiment is mounted onan interior ceiling structure of the vehicle 100. The front cabin camera204 is mounted in any location to position at least the entire frontoccupant seats 140, the dashboard 144, the center console 156, and atleast a portion of the floor 236 within the field of view 232. Theimages within the field of view 232, as generated by the front cabincamera 204, are included in the cabin image data 228.

The rear cabin camera 204 (i.e. the cabin camera 204 positioned farthestfrom the dashboard 144 in FIG. 1) defines a field of view 240 (FIG. 1)that includes at least the entire rear occupant seats 140 and at least aportion of the floor 236. The rear cabin camera 204, in one embodiment,is mounted on the interior ceiling structure of the cabin 132. The rearcabin camera 204 is mounted in any location to position at least theentire rear occupant seats 140 and at least a portion of the floor 236within the field of view 240. The images within the field of view 240,as generated by the rear cabin camera 240, are included in the cabinimage data 228.

In some embodiments, the litter sensing system 104 includes only onecabin camera 204 that is positioned to generate cabin image data 228 ofthe entire cabin 236 including all of the occupant seats 140. In otherembodiments, the litter sensing system 104 includes three or more of thecabin cameras 204. For example, in a passenger van, minivan, or othervehicle having three or more rows of seats 140, each row of seats 140may include a corresponding cabin camera 204. In yet another embodiment,each individual seat 140 of the vehicle 100 is monitored by anindividual cabin camera 204. The litter sensing system 104 includes anynumber of the cabin cameras 204 as may be used to monitor the occupants108 and/or seats 140 of the corresponding vehicle 100.

The mirror cameras 208 are digital imaging devices configured togenerate window image data 244. In one embodiment, the mirror cameras208 are surround-view cameras positioned to generate window image data244 of the object 112 as the object 112 is thrown out of the windowopening 116. The left mirror camera 208 is mounted on the left sidemirror housing 160, and the right mirror camera 208 is mounted on theopposite right side mirror housing 160. The mirror cameras 208 may beconfigured as visible light cameras, thermal cameras, and/or infraredcameras.

As shown in FIGS. 1 and 2, the left mirror camera 208 defines a field ofview 248 that includes the left side of the vehicle 100 including theentire window opening 116 of the front door 124 and the entire windowopening 116 of the rear door 124. The right mirror camera 208 defines afield of view 252 that includes the right side of the vehicle 100including the entirety of the corresponding other window openings 116.The fields of view 248, 252 of the mirror cameras 208, in at least oneembodiment, also extend above a roof 256 (FIG. 2) of the vehicle 100 andto the ground on which the vehicle 100 is positioned. Since the fieldsof view 248, 252 of the mirror cameras 208 include the entirety of eachwindow opening 116, any object 112 passed through one of the windowopenings 116 is detected by at least one of the mirror cameras 208. Theimages within the fields of view 248, 252 of the mirror cameras 208 areincluded in the window image data 244.

The transceiver 212, which is also referred to as a wireless transmitterand receiver, is configured to wirelessly transmit data from the vehicle100 to another electronic device (not shown) and to wirelessly receivedata from another electronic device via the Internet, for example. Thus,the transceiver 212 operably connects the vehicle 100 to the Internetand to other electronic devices. In other embodiments, the transceiver212 sends and receives data using a cellular network, a wireless localarea network (“Wi-Fi”), a personal area network, and/or any otherwireless network. Accordingly, the transceiver 212 is compatible withany desired wireless communication standard or protocol including, butnot limited to, Near Field Communication (“NFC”), IEEE 802.11, IEEE802.15.1 (“Bluetooth®”), Global System for Mobiles (“GSM”), and CodeDivision Multiple Access (“CDMA”).

The memory 216 is an electronic storage device that is configured tostore at least the cabin image data 228, the window image data 244, andprogram instructions 260 for operating the litter sensing system 104.The memory 216 is also referred to herein as a non-transient computerreadable medium.

The memory 216 is further configured to store window state data 264,door state data 268, and speed data 270. The window state data 264 aregenerated by the controller 220 based on signals received from the ECU164, which is electrically connected to the window state sensors 170.The window state data 164 indicates which of the window panels 128 arein the open window state and which of the window panels 128 are in theclosed window state at any particular time. Additionally oralternatively, the window state data 264 are generated by the ECU 164and transmitted to the controller 220 for storage in the memory 216.

The door state data 268 are generated by the controller 220 based onsignals received from the ECU 164, which is electrically connected tothe door state sensors 174. The door state data 268 indicates which ofthe doors 124 are in the open door state and which of the doors 124 arein the closed door state at any particular time. Additionally oralternatively, the door state data 268 are generated by the ECU 164 andtransmitted to the controller 220 for storage in the memory 216.

The speed data 270 correspond to a ground speed of the vehicle 100. Thespeed data 270 in one embodiment is generated by the controller 220based on the signal from the speed sensor 178, as received by thecontroller 220 from the ECU 164. In another embodiment, the ECU 164generates the speed data 270 and transmits the speed data 270 to thecontroller 220 for storage in the memory 216. The speed data 270, insome embodiments, also include data corresponding to a drive state ofthe vehicle 100. For example, the drive state may include a drive modeand a non-drive mode, and corresponding information is included in thespeed data 270. The speed data 270 may indicate that the vehicle 100 istraveling at zero miles per hour in the park mode, traveling at tenmiles per hour in the drive mode, or is stationary in the drive mode.

The drive mode is a drive state of the vehicle 100 in which the vehicle100 is capable of being moved by the engine and/or the electric motor.In a passenger vehicle 100 the drive mode corresponds to putting thetransmission in “D” for forward movement or in “R” for reverse movement.In an autonomous vehicle 100, the drive mode corresponds to any drivestate of the vehicle 100 in which the vehicle 100 may use the engineand/or the motor to move the vehicle 100. The drive mode is contrastedwith the non-drive mode, which is typically referred to as a neutralmode “N” or a park mode “P”. The vehicle 100 is not drivable (manuallyor autonomously) in the non-drive mode.

The memory 216 also stores ejected object data 272 and littering eventdata 276. The ejected object data 272 are generated by the controller220 based on the window image data 244. The ejected object data 272include image data of the object 112 ejected, thrown, or dropped fromthe cabin 132 through one of the vehicle window openings 116. As shownin FIG. 2, the object 112 has been ejected from the cabin 132 by theoccupant 108 through the window opening 116. The object 112 is withinthe field of view 248 of the mirror camera 208 and corresponding imagedata of the object 112 are included in the window image data 244. Basedon processing performed by the controller 220, data corresponding to theejected object 112 is identified, isolated, derived, and/or classifiedin the window image data 244 and is stored separately in the memory 216as the ejected object data 272. That is, the window image data 244includes all of the image data generated by the mirror cameras 208, andthe ejected object data 272 includes the window image data 244 thatincludes images of the ejected object 112, as automatically identifiedby the controller 220.

The littering event data 276 stored in the memory 216 includes at leasta portion of the ejected object data 272 and at least one of a date, atime, and a location at which the ejected objected 112 was ejected fromthe vehicle 100. The location is determined from an onboard globalpositioning system (not shown) of the vehicle 100, for example. Thelittering event data 276 when rendered on a suitable display (such as acomputer monitor), is useful for a person to analyze to determine if alittering event has actually taken place or if the alleged object 112was road debris or other non-litter elements.

In addition to the above, the memory 216 also stores occupant data 280,throwing motion data 284, and incoming object data 288. The occupantdata 280 includes at least one of a name, address, and account number ofthe occupant 108. In another embodiment, the occupant data 280 includeseat position data based on a position(s) of each occupant seat 140, andthe occupant data 280 do not include identification data of the occupant108. Instead, the vehicle sensing system 104 transmits the occupant data280 to a fleet operator, for example, of the vehicle 100, and the fleetoperator associates the occupant data 280 with personal identificationdata of the occupant 108. Thus, in at least some embodiments, personalidentification data of the occupant 108 are not stored by the vehiclesensing system 104.

The throwing motion data 284 are identified, isolated, derived, and/orclassified from the cabin image data 228 by the controller 220 andcorrespond to images of an arm or a hand of the occupant 108 movingtoward the vehicle window opening 116 in a tossing or a throwing motion.Specifically, the throwing motion data 284 are a portion of the cabinimage data 228 that have been identified by the controller 220 asincluding image data corresponding to an occupant 108 making a hand orarm movement that is consistent with throwing the object 112 out of thewindow opening 116. That is, the cabin image data 228 includes all ofthe image data generated by the cabin cameras 204, and the throwingmotion data 284 includes the cabin image data 228 that includes imagesof arm and hand movements toward the window openings 116, asautomatically identified by the controller 220. Identification and/orgeneration of the throwing motion data 284 is, at times, a precursor toa littering event, because the occupant 108 may be moving the object 112to be ejected toward the window opening 116.

The incoming object data 288 is another portion of the cabin image data228 corresponding to objects 112 moved into the cabin 132. For example,when the occupant 108 enters the vehicle 100 with a drink container(represented by the object 112), the cabin camera 204 generatescorresponding cabin image data 228. The controller 220 processes thecabin image data 228 to identify, isolate, derive, and/or classify imagedata corresponding to elements (i.e. the object 112) brought into thecabin 132, according to the method 400 described below.

The controller 220 of the vehicle sensing system 104 is configured toexecute the program instruction data 260 in order to operate the sensingsystem 104. The controller 220 is provided as at least onemicrocontroller and/or microprocessor. In one embodiment, the controller220 is electrically connected to the ECU 164 by the CAN bus to receiveone or more of the window state data 264, the door state data 268, andthe speed data 270.

In operation, the system 104 is configured to implement the method 400illustrated by the flowchart of FIG. 4. In this exemplary embodiment,the vehicle 100 is a shared autonomous vehicle that is configured todrive autonomously to the location of an occupant 108, then upon theoccupant 108 entering the vehicle 100, transport autonomously theoccupant 108 to a desired location using the public roadway network. Theoccupant 108 may engage the services of the vehicle 100 using asmartphone application (i.e. an “app”), for example. The occupant 108 isalso referred to herein as a passenger, a user, an operator, or aperson. In other embodiments, the vehicle 100 is any type of passengervehicle, as described above, and may be occupant 108 controlled.

In block 404 of the method 400, the vehicle 100 is stationary and thelitter sensing system 104 detects the objects 112 entering and exitingthe vehicle 100 through the open doors 124 using the cabin cameras 204.The system 104 determines that the vehicle 100 is stationary based onthe speed data 270. Moreover, the system 104 determines which of thedoors 124 is open (i.e. is in the open door state) based on the doorstate data 268. The system 104, in one embodiment, generates theincoming object data 288 only when at least one door 124 is in the opendoor state. The system 104 does not generate the incoming object data288 when all of the doors 124 are closed. The system 104 generates thecabin image data 228 with the cabin cameras 204, which includes imagesof the occupant 108 and the object 112 entering cabin 132. The occupant108 places the object 112, which in this example is a drink container ora coffee cup, in a cup holder, for example. The object 112 is in thefield of view 232 of the front cabin camera 204.

The controller 220 processes the cabin image data 228 to generate theincoming object data 288, which, in this example, corresponds to theobject 112. In one embodiment, the controller 220 uses computer visionobject detection techniques to identify common objects in the cabinimage data 228. The incoming object data 288 are a representation ofeach object 112 detected by the sensing system 104 as being brought intothe cabin 132 through the open doors 124. In one embodiment, the system104 distinguishes objects 112 from occupants 108 and identifies anythingthat is not an occupant 108 and is not the vehicle 100 as an incomingobject 112.

At block 404, the system 104 may detect objects 112 exiting the vehicle100 through the open doors 124 based on the cabin image data 228. Whenthe vehicle 100 is in the non-drive mode (and is stationary) and anobject 112 is detected as moving from the cabin 132 out of the fields ofview 232, 240 of the cabin cameras 204 through an open door 124 or thewindow opening 116, the object 112 is removed from the incoming objectdata 288. The movement of an object 112 through an open door 124 is notprocessed by the system 104 as a littering event and does not result inthe generation of the littering event data 276. This is because whenobjects are moved through the open doors 124, it is far more likely thatthe occupants 108 are unloading objects 112 from the vehicle 100 ratherthan littering or improperly disposing of the objects 112.

Next, in block 408 of the method 400, the controller 220 determineswhether or not the vehicle 100 is in the drive mode and/or is moving. Ifthe controller 220 determines that the vehicle 100 is in the drive mode,then the method 400 continues to block 412. The system 104 determinesthe state of the vehicle 100 and whether or not the vehicle 100 is inmotion or stationary based on the speed data 270.

At block 412, the system 104 determines if any of the movable windowpanels 128, including a movable sunroof and/or moonroof movable windowpanel 128, are in the open window state based on the window state data264. If the system 104 determines that all of the window panels 128 areclosed, then the method 400 returns to block 408 to monitor the state ofthe vehicle 100 based on the speed data 270. The system 104 determinesthat a littering event cannot occur with the vehicle 100 in the drivemode and with all of the window panels 128 in the closed window state,because there are no openings through which to eject the object 112. Ifthe system 104 determines that at least one window panel 128 is in theopen window state, then the method 400 progresses to block 416.

At block 416, the system 104 monitors the position of the objects 112 aswell as the behavior and the movements of the occupants 108 because alittering event could potentially occur. That is, at block 416, thevehicle 100 is in the drive mode and at least one window panel 128 is inthe open window state. Thus, an opening exists through which theoccupant 108 may eject the object 112 from the vehicle 100 and theoccupant 108 is unlikely to be loading or unloading the vehicle 100 withthe vehicle in the drive mode. The object 112 is representative of anyelement within the vehicle 100 that could be thrown out of a windowopening 116 and includes personal items, trash, food, and the like.

In one embodiment, at block 416, the controller 220 compares the windowimage data 244 to the incoming object data 288 to determine the ejectedobject data 272. In particular, the controller 220 stores image data ofeach object 112 that is brought into the cabin 132 by processing thecabin image data 228 and by generating the incoming object data 288.Then, at block 416, the controller 220 processes the window image data244 for data corresponding to the objects 112 of the incoming objectdata 288. If any objects 112 of the incoming object data 288 aredetected in the window image data 244, the controller 220 determinesthat the corresponding object 112 has been ejected from the cabin 132through the window opening 116.

As an example, when the object 112 is loaded into the cabin 132, thecontroller 220 generates incoming object data 288 corresponding toimages of the object 112. Then, when the vehicle 100 is in the drivemode, the controller 220 processes the window image data 244 usingcomputer vision techniques to search for corresponding images of theobject 112 in the window image data 244. If the controller 220identifies a match between the object 112 in the window image data 244and the object 112 in the incoming object data 288, then the ejectedobject data 272 is generated and includes images of the object 112 thatwas ejected from the vehicle 100 from the window image data 244. In oneembodiment, in order for the object 112 to appear and to be detected inthe window image data 244 it must have been ejected from the vehicle100, because the fields of view 248, 252 of the mirror cameras 208 arelocated only outside of the cabin 132 of the vehicle 100 and do notinclude any portion of the cabin 132.

In another embodiment, the controller 220 processes only the windowimage data 244 to generate the ejected object data 272 withoutgenerating the incoming object data 288. Thus, in some embodiments, theincoming object data 288 is not required. The window image data 244 isgenerated only when the system 104 detects that a window panel 128 is inthe open window state. In this embodiment, the controller 220 usescomputer vision techniques to detect the object 112 in a correspondingfield of view 248, 252 of the mirror cameras 208. Much of the imagesgenerated as the window image data 244 are static images of the side ofthe vehicle 100. Thus, to detect the object 112 in the window image data244, a subtractive approach may be employed by the controller 228 toidentify differences between images of the side of the vehicle 100 withno objects 112 present in the fields of view 248, 252, and images of theside of the vehicle 100 that include an object 112 in at least one ofthe fields of view 248, 252.

Moreover, additional or alternative image processing techniques may beused to identify the object 112 in the window image data 244. Forexample, the controller 220 processes the window image data 244 for anyelement that tracks front-to-back along the side of the vehicle 100, aswould the object 112 when it is ejected from the vehicle 100. When suchimage data is detected, the corresponding window image data 244 is savedto the ejected object data 272.

The controller 220, in one embodiment, is additionally or alternativelyconfigured to identify as ejected object data 272, objects 112 in thewindow image data 244 that are detected as bouncing on the road orfalling from the vehicle 100 and hitting the road. Such a track in thewindow image data 244 has a downward trajectory along the exterior ofthe door 124 and is indicative of an ejected object 112.

When processing the window image data 244, the controller 220, in atleast one embodiment, is configured to ignore image data correspondingto the hand or the arm of the occupant 108 resting on the window opening116. Such an approach reduces false-positive littering event data 276.

Additionally or alternatively, at block 416 the system 104 uses thethrowing motion data 284 as a trigger for starting to generate and toprocess the window image data 244. For example, when ejecting the object112 from the cabin 132, the occupant 108 grabs the object 112 and movesthe object 112 towards the window opening 116 with a throwing or atossing motion. Such a movement of the occupant 108 and the object 112is depicted in FIG. 1. The controller 220 detects the throwing motion asthe throwing motion data 284 from the cabin image data 228. Upon thedetection of the throwing motion data 284, the controller 220 startsgenerating the window image data 244 because an ejected object 112 maybe forthcoming. In this manner, the system 104 does not continuouslygenerate the window image data 244, but instead only generates thewindow image data 244 after the detection of throwing motion data 284and when at least one window panel 128 is in the open window state. Suchan approach also tends to reduce the amount of false-positive litteringevent data 276.

In a further embodiment at block 416, the controller 220 processes thecabin image data 228 to detect if any object 112 is moved toward awindow opening 116. Accordingly, instead of or in addition to detectingthe throwing motion data 284, the position of each object 112 within thecabin 132 is monitored and determined by the controller 228. When anyobject 112 is determined to have been moved sufficiently toward or nearthe window opening 116, then the controller 220 starts to generate thewindow image data 244 because an ejected object 112 may be forthcoming.In one embodiment, when an object 112 detected in the cabin image data228 is moved within a distance threshold from the window opening 116,then the controller 220 begins to generate the window image data 244.The distance threshold is from two to twelve inches, for example.

In yet another embodiment at block 416 of the method 400, the controller220 only generates the window image data 244 using a selected one of theplurality of mirror cameras 20. For example, if only the front leftwindow panel 128 is in the open window state (as shown in FIG. 2), thenthe controller 220 generates the window image data 244 with only theleft mirror camera 208 and does not generate the window image data 244with the right mirror camera 208 because an object 112 cannot be ejectedfrom the right side of the vehicle 100 due to all of the window panels128 being in the closed window state.

Moving to block 420 of the method 400, the controller 220 determines iflittering has occurred and generates the littering event data 276. Inparticular, the system 104 generates the littering event data 276 byassociating the ejected object data 272 with at least one of a time, adata, and a location of the vehicle 100. Moreover, the littering eventdata 276 includes the occupant data 280, which may include theoperator's account number, name, residential address, email address,and/or workplace. Thus, the controller 220 associates the ejected objectdata 272 with the occupant data 280 when generating the littering eventdata 276. In some embodiments, frames of the cabin image data 228 areincluded in the littering event data 276 to connect further the ejectedobject 112 with a specific occupant 108 included in the cabin image data228. Specifically, images of the occupant's face may be saved andcataloged in the littering event data 276 in order to identify theoccupant 108. In another embodiment, the occupant data 280 do notinclude personal identification data of the occupant 108 and theoccupant data 280 include only vehicle 100 state data, such as seatposition data, for example.

Also at block 420, in one embodiment, the system 104 uses the displayscreen 148 and/or the speaker 152 to alert the occupant 108 thatlittering has been detected. When the littering event data 276 isgenerated, the controller 220 sends the ejected object data 272 and/orthe littering event data 276 to the display screen 148 for display tothe occupant 108. The display screen 148, for example, shows theoccupant 108 images of the ejected object 112 and a message may bebroadcast on the speaker 152 requesting the occupant 108 to cease alllittering activity. The display screen 148 and/or the audio message mayfurther inform the occupant 108 that they will face a service charge dueto the ejection of the object 112 from the vehicle 100. The displayscreen 148 and/or the audio message may additionally or alternativelyinform the occupant 108 that they can no longer utilize the vehicle 100for future transportation needs so as prevent occupants 108 that areknown to litter from littering again. In further embodiments, whenlitter is detected, an account of the occupant 108 may be revoked orchanged until appropriate steps have been taken to rectify the detectedlittering event, such as paying the service charge, cleaning up thelitter, and/or performing a certain number of community service hours.

The littering event data 276, in one embodiment, is transmitted to aremote location by the transceiver 212 for processing and storage. Forexample, when littering event data 276 is received at the remotelocation a human analyst may review the ejected object data 272 todetermine if actual littering has occurred or if the littering eventdata 276 was based on a false detection.

With reference still to FIG. 4, if at block 408 the controller 220determines that the vehicle 100 is not in the drive mode (i.e. is in thenon-drive mode), then the method 400 continues to block 424. At block424, the system 104 again determines if any of the movable window panels128, including a movable sunroof and/or moonroof movable window panel128, are in the open window state based on the window state data 264. Ifthe system 104 determines that all of the window panels 128 are in theclosed window state, then the method 400 returns to block 404 to detectthe objects 112 entering and exiting the vehicle 100 through the opendoors 124 (if any) using the cabin cameras 204. If the system 104determines that at least one window panel 128 is in the open windowstate in block 424, then the method 400 progresses to block 428.

At block 428, the controller 220 process the window image data 244 toidentify as littering, objects 112 that are ejected through the windowopenings 116 and that contact the ground on which the vehicle 100 ispositioned. The corresponding window image data 244 is stored as thelittering event data 276 in the manner described above. Thereafter, themethod 400 continues to block 408 to monitor if the vehicle 100 is inthe drive mode.

The processing steps that occur at block 428 account for numerous usecases including drive-through businesses, automatic teller machines(“ATM”), and passing items to people outside of the vehicle 100, andserve to prevent false-positive litter detection by the system 104.Specifically, in the use case of a drive-through business, the occupant108 of the vehicle 100 passes payment to a person outside of the vehicle100 through the window opening 116 and in exchange receives items fromthe person outside of the vehicle 100 through the window opening 116.Neither the payment nor the received items contact the ground on whichthe vehicle 100 is located and accordingly, the movement of the paymentand the received items is not littering and should not be detected aslittering by the system 104. Similarly, when using a trashcan locatedoutside of the vehicle 100, the occupant 108 moves an object through thewindow opening 116 and drops the object into the trashcan. The discardedobject does not contact the ground and is not littering. In a litteringevent, however, the occupant 108 passes or throws the object 112 throughthe window opening 116 and the object 112 falls to the ground. Theejected object 112 falls from the window opening 116 and strikes theground. The controller 220, therefore, is configured to process thewindow image data 244 to detect data corresponding to objects 112 thatpass through the window opening 116 and then fall to the ground, sincesuch data are very likely to correspond to littering events. Thecorresponding window image data 244 are included in the littering eventdata 276.

Prior to or during the processing step of block 428, the method 400 mayalso include the processing step of block 416 to monitor the position ofthe detected objects 112 and/or to monitor the behavior of the occupants108 within the vehicle 100.

While the disclosure has been illustrated and described in detail in thedrawings and foregoing description, the same should be considered asillustrative and not restrictive in character. It is understood thatonly the preferred embodiments have been presented and that all changes,modifications, and further applications that come within the spirit ofthe disclosure are desired to be protected.

What is claimed is:
 1. A method for detecting objects ejected from acabin of a vehicle, comprising: generating window image data of avehicle window opening of the vehicle with a camera mounted on thevehicle; processing the window image data with a controller operablyconnected to the camera to generate ejected object data corresponding toat least one object ejected from the cabin through the vehicle windowopening; and associating the ejected object data with occupant data ofan occupant of the vehicle with the controller.
 2. The method as claimedin claim 1, further comprising: detecting a state of a movable windowpanel that is movable within the vehicle window opening with at leastone window state sensor of the vehicle, the detected state including anopen window state and a closed window state; and generating the windowimage data only when the movable window panel is in the open windowstate.
 3. The method as claimed in claim 2, wherein the camera mountedon the vehicle is a first camera, and the method further comprises:generating cabin image data of the cabin of the vehicle with a secondcamera mounted inside the cabin, the cabin image data at least partiallybased on movements of the occupant within the cabin; processing thecabin image data with the controller to detect throwing motion datacorresponding to a motion of an arm or a hand of the occupant toward thevehicle window opening; and generating the window image data only whenthe movable window panel is in the open window state and the throwingmotion data are detected.
 4. The method as claimed in claim 1, furthercomprising: generating the window image data with the camera mountedoutside of the cabin.
 5. The method as claimed in claim 4, wherein thecamera is mounted directly on a side mirror housing of the vehicle. 6.The method as claimed in claim 1, further comprising; generatinglittering event data based on the generated ejected object data and theoccupant data with the controller, wherein the littering event dataincludes image data of the ejected object and at least one of a date, atime, and a location at which the ejected objected was ejected from thevehicle.
 7. The method as claimed in claim 1, wherein the camera mountedon the vehicle is a first camera, and the method further comprises:generating incoming object data with a second camera mounted in thecabin, the incoming object data corresponding to incoming objects placedin the cabin by the occupant; and identifying the incoming objects inthe window image data with the controller as the at least one object isejected from the cabin, wherein the first camera is mounted outside ofthe cabin.
 8. The method as claimed in claim 7, further comprising:detecting a state of at least one door of the vehicle, the detectedstate including an open door state and a closed door state; andgenerating the incoming object data only when the at least one door isin the open door state.
 9. The method as claimed in claim 1, wherein theejected object data include image data of the at least one object. 10.The method as claimed in claim 9, further comprising: assessing aservice charge to the occupant of the vehicle.
 11. A method fordetecting objects ejected from a cabin of a vehicle, comprising:generating window image data of a vehicle window opening when acorresponding movable window panel is in an open position; processingthe window image data to generate ejected object data corresponding toobjects ejected from the cabin through the vehicle window opening;generating littering event data based on the generated ejected objectdata; and associating the littering event data with an occupant of thevehicle.
 12. The method as claimed in claim 11, further comprising:detecting a state of the movable window panel, the detected stateincluding an open window state corresponding to the open position and aclosed window state corresponding to a closed position of the movablewindow panel; and generating the window image data only when the movablewindow panel is in the open window state.
 13. The method as claimed inclaim 12, further comprising: generating the window image data with afirst camera mounted outside the cabin; generating cabin image data ofthe cabin of the vehicle with a second camera mounted inside the cabin,the cabin image data at least partially based on movements of theoccupant within the cabin; processing the cabin image data to detectthrowing motion data corresponding to a motion of an arm or a hand ofthe occupant toward the vehicle window opening; and generating thewindow image data only when the movable window panel is in the openwindow state and the throwing motion data are detected.
 14. The methodas claimed in claim 11, further comprising: generating the window imagedata with a camera mounted outside of the cabin.
 15. The method asclaimed in claim 14, wherein the camera is mounted directly on a sidemirror housing of the vehicle.
 16. A system for detecting objectsejected from a cabin of a vehicle, comprising: a camera mounted on thevehicle and configured to generate window image data of a vehicle windowopening; and a controller operably connected to the camera to receivethe window image data, the controller configured (i) to process thewindow image data to generate ejected object data corresponding to atleast one object ejected from the cabin through the vehicle windowopening, and (ii) to associate the ejected object data with occupantdata of an occupant of the vehicle.
 17. The system as claimed in claim16, wherein the camera mounted on the vehicle is a first camera, furthercomprising: a second camera mounted on the vehicle and configured togenerate cabin image data of the cabin of the vehicle, the cabin imagedata at least partially based on movements of the occupant within thecabin, wherein at least one window state sensor of the vehicle isconfigured to detect a state of a movable window panel that is movablewithin the vehicle window opening, the detected state including an openwindow state and a closed window state, and wherein the controller isfurther configured (i) to process the cabin image data to detectthrowing motion data corresponding to a motion of an arm or a hand ofthe occupant toward the vehicle window opening, and (ii) to generate thewindow image data only when the movable window panel is in the openwindow state and the throwing motion data are detected.
 18. The systemas claimed in claim 17, wherein: the first camera is located within thecabin, and the second camera is mounted directly on a side mirrorhousing of the vehicle.
 19. The system as claimed in claim 16, wherein:the controller is configured to generate and to send the ejected objectdata to a display screen of the vehicle, and the display screen isoperably connected to the controller and is configured to display amessage to the occupant based on the ejected object data.
 20. The systemas claimed in claim 19, wherein the message instructs the occupant tostop littering.